Feature/neural nets factory and cookbook #4386

shubham808 · 2018-07-27T13:43:55Z

No description provided.

karlnapf · 2018-07-27T14:43:08Z

Looks good, how does it work?

karlnapf · 2018-07-27T14:43:50Z

examples/meta/src/neural_nets/convolutional_net_classification.sg

+
+#![train_and_apply]
+network.train(features_train)
+Labels labels_predict = network.apply_multiclass(features_train)


karlnapf · 2018-07-27T15:34:01Z

examples/meta/src/neural_nets/feedforward_net_classification.sg

 #![add_layers]

 #![create_instance]
-NeuralNetwork network(all_layers)
+NeuralNetwork network = neural_network("NeuralNetwork", labels=labels_train)


Neural network is a machine, so this could be instantiated using machine
This means that the below methods set_layers, quick_connect, and initialize_neural_network all would have to be changed.

set_layers can be done via put (already now!) just needs a dispatcher for the CDynamicObjectArray in put.

quick_connect, and initialize_neural_network could just be called automatically inside the train_machine method which should be a minor refactor

karlnapf · 2018-07-27T15:34:18Z

examples/meta/src/neural_nets/feedforward_net_classification.sg

 network.train(features_train)
-BinaryLabels labels_predict = network.apply_binary(features_test)
+Labels labels_predict = network.apply_binary(features_test)


karlnapf · 2018-07-27T15:35:13Z

src/shogun/util/factory.h

@@ -34,6 +36,8 @@ namespace shogun
 	CECOCEncoder* ecoc_encoder(const std::string& name);
 	CECOCDecoder* ecoc_decoder(const std::string& name);
 	CTransformer* transformer(const std::string& name);
+	CNeuralLayer* neural_layer(const std::string& name);
+	CNeuralNetwork* neural_network(const std::string& name);


Would rather not have a network factory, but instead use machine

karlnapf

Looks good! I think this is a good basis. Just need the refactoring to clean things up a bit

karlnapf · 2018-07-27T15:40:55Z

Please also update to only using 2 classes (and then try less data and also pls post the runtime of the meta example)

shubham808 · 2018-07-28T11:59:36Z

src/shogun/neuralnets/NeuralNetwork.cpp

+	{
+		quick_connect();
+	}
+	if (!m_is_initialized)


this is cleaner since it will avoid multiple calls to initialize_neural_network

although ....
why would there be multiple calls? Due to multiple "train" calls?

also would need to make sure that these things are set to false if the network architecture is changed.
So actually, it might be that always calling these might be a better idea. The cost is neglectible.
Thoughts?

karlnapf · 2018-07-29T00:30:06Z

examples/meta/src/neural_nets/convolutional_net_classification.sg

+#![add_layers]
+
+#![create_instance]
+Machine network = machine("NeuralNetwork", labels=labels_train, max_num_epochs=4, epsilon=0.0, optimization_method=enum ENNOptimizationMethod.NNOM_GRADIENT_DESCENT, gd_learning_rate=0.01, gd_mini_batch_size=3, max_norm=1.0, dropout_input=0.5, layers=layers_conv)


karlnapf · 2018-07-29T00:30:35Z

examples/meta/src/neural_nets/feedforward_net_classification.sg

-layers.softmax(2)
-DynamicObjectArray all_layers = layers.done()
+DynamicObjectArray all_layers()
+NeuralLayer input = neural_layer("NeuralInputLayer", num_neurons=num_feats)


I wonder, could we go with the simpler name layer?

karlnapf · 2018-07-29T00:31:03Z

src/shogun/base/base_types.h

@@ -31,7 +31,8 @@ namespace shogun
 	                    std::is_same<CLabels, T>::value ||
 	                    std::is_same<CECOCEncoder, T>::value ||
 	                    std::is_same<CECOCDecoder, T>::value ||
-	                    std::is_same<CMulticlassStrategy, T>::value>
+	                    std::is_same<CMulticlassStrategy, T>::value ||
+	                    std::is_same<CDynamicObjectArray, T>::value>


exactly like this!

karlnapf · 2018-07-29T12:34:40Z

examples/meta/src/neural_nets/feedforward_net_regression.sg

-layers.linear(1)
-DynamicObjectArray all_layers = layers.done()
+int num_feats = features_train.get_int("num_features")
+DynamicObjectArray all_layers()


btw instead of using the array type here explicitly, we should be able to use SGObject::add, which allows to append to an internal array.
The call should be machine.add("layers", input); etc. This means you would first create the network instance, and only then add the layers.
If this works, we dont need to add the array type to shogun's base classes, which I would prefer

see the multiple kernel learning example where the same thing is done

karlnapf

nice!
if we can get rid of the array type, that would be better, Sorry I just only realised this.
The other thing is that we want to use 2 classes only

shubham808 · 2018-07-31T06:20:01Z

src/shogun/neuralnets/NeuralNetwork.cpp

@@ -229,6 +234,13 @@ CDenseFeatures< float64_t >* CNeuralNetwork::transform(

 bool CNeuralNetwork::train_machine(CFeatures* data)
 {
+	if (m_auto_quick_initialize)


i think this is cleaner since there are no states now :) the code will work like it did before

karlnapf · 2018-07-31T08:44:18Z

src/shogun/neuralnets/NeuralNetwork.cpp

+	SG_ADD(
+	    &m_layers, "layers", "DynamicObjectArray of NeuralNetwork objects",
+	    MS_NOT_AVAILABLE);
+	SG_ADD(&m_auto_quick_initialize, "auto_quick_initialize", "auto_quick_initialize", MS_NOT_AVAILABLE);


This needs a better description.
"Whether the network is automatically connected and initialized on training."

karlnapf · 2018-07-31T08:44:34Z

src/shogun/neuralnets/NeuralNetwork.h

+	/** True if the network layers are to be quick connected and initialized
+	 * initial value is False
+	 */
+	bool m_auto_quick_initialize;


pls refer to the doxygen methods that are called here

karlnapf · 2018-07-31T08:45:05Z

examples/meta/src/neural_nets/feedforward_net_classification.sg

-#![set_parameters]
+#![add_layers]
+NeuralLayer input = layer("NeuralInputLayer", num_neurons=num_feats)
+network.add("layers", input)


yes this is much nicer. Should have thought about it straight away :)

karlnapf · 2018-07-31T08:45:23Z

examples/meta/src/neural_nets/feedforward_net_classification.sg

-network.set_gd_learning_rate(0.1)
-network.set_gd_momentum(0.9)
-#![set_parameters]
+#![add_layers]


if you change these references you will need to change the cookbook ones as well! It refers to these snippets!

right i will edit the cookbook too :)

karlnapf · 2018-07-31T08:46:28Z

examples/meta/src/neural_nets/feedforward_net_regression.sg

-layers.input(dimensions)
-layers.rectified_linear(20)
-layers.linear(1)
-DynamicObjectArray all_layers = layers.done()


Is this taken care of?

ah yes it just returns the DynamicObjectArray so if we set that instead using add, all should be good

karlnapf · 2018-07-31T08:48:34Z

src/shogun/neuralnets/NeuralNetwork.cpp

-
+	m_auto_quick_initialize = false;
+	m_sigma = 0.01f;
+	m_layers=new CDynamicObjectArray();


you need to unref this in the destructor

karlnapf · 2018-07-31T08:49:21Z

src/shogun/neuralnets/NeuralNetwork.cpp

@@ -772,7 +784,11 @@ void CNeuralNetwork::init()
 	m_lbfgs_temp_inputs = NULL;
 	m_lbfgs_temp_targets = NULL;
 	m_is_training = false;
-
+	m_auto_quick_initialize = false;
+	m_sigma = 0.01f;


the f is not necessary

karlnapf

Apart from some minor bits, this is good!

karlnapf · 2018-07-31T18:34:46Z

examples/meta/src/neural_nets/convolutional_net_classification.sg

-File f_feats_test = csv_file("../../data/fm_test_mnist_16x16.dat")
-File f_labels_train = csv_file("../../data/labels_train_mnist_16x16.dat")
-File f_labels_test = csv_file("../../data/labels_test_mnist_16x16.dat")
+File f_feats_train = csv_file("../../data/classifier_3class_256d_linear_features_train.dat")


I am confused by the "linear" in the filename.
Also I think it should still say "mnist"
What about mnist_3class_256d_features_train.dat

yes that is a better name :)

karlnapf · 2018-07-31T18:35:16Z

Happy to merge after the filename change

karlnapf · 2018-07-31T21:54:26Z

Tada :)

* convolutional net meta example * data update * cnn integration test * use add api for layers * rename factory and initialize layers * convolutional layer cookbook and other updates

shubham808 added 3 commits July 27, 2018 19:11

neural nets factory

25506fb

convolutional net meta example

258e25b

data update

9ae858a

karlnapf reviewed Jul 27, 2018

View reviewed changes

karlnapf requested changes Jul 27, 2018

View reviewed changes

shubham808 added 3 commits July 28, 2018 12:21

use apply

42e4c0f

remove neural_network factory and use machine

96a3855

cnn integration test

5378b9e

shubham808 commented Jul 28, 2018

View reviewed changes

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch 2 times, most recently from b8f7455 to f680424 Compare July 28, 2018 19:11

karlnapf reviewed Jul 29, 2018

View reviewed changes

karlnapf requested changes Jul 29, 2018

View reviewed changes

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch 2 times, most recently from c6ce549 to eb9290a Compare July 31, 2018 01:00

use add api for layers

db95fb4

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch from eb9290a to db95fb4 Compare July 31, 2018 01:35

rename factory and initialize layers

11b3922

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch from 2817b53 to 11b3922 Compare July 31, 2018 04:12

shubham808 commented Jul 31, 2018

View reviewed changes

karlnapf reviewed Jul 31, 2018

View reviewed changes

karlnapf requested changes Jul 31, 2018

View reviewed changes

use 3 classes

77aa3ff

karlnapf reviewed Jul 31, 2018

View reviewed changes

convolutional layer cookbook and other updates

a6ddbae

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch 2 times, most recently from b1cbac8 to 9aae0bb Compare July 31, 2018 19:29

rename data files

599d431

shubham808 force-pushed the feature/neural_nets_factory_and_cookbook branch from 9aae0bb to 599d431 Compare July 31, 2018 19:30

karlnapf merged commit aac28bd into shogun-toolbox:develop Jul 31, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/neural nets factory and cookbook #4386

Feature/neural nets factory and cookbook #4386

shubham808 commented Jul 27, 2018

karlnapf commented Jul 27, 2018

karlnapf Jul 27, 2018

karlnapf Jul 27, 2018

karlnapf Jul 27, 2018

karlnapf Jul 27, 2018

karlnapf left a comment

karlnapf commented Jul 27, 2018

shubham808 Jul 28, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf Jul 29, 2018

karlnapf left a comment

shubham808 Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

shubham808 Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf Jul 31, 2018

karlnapf left a comment

karlnapf Jul 31, 2018

shubham808 Jul 31, 2018

karlnapf commented Jul 31, 2018

karlnapf commented Jul 31, 2018

Feature/neural nets factory and cookbook #4386

Feature/neural nets factory and cookbook #4386

Conversation

shubham808 commented Jul 27, 2018

karlnapf commented Jul 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlnapf left a comment

Choose a reason for hiding this comment

karlnapf commented Jul 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlnapf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlnapf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlnapf commented Jul 31, 2018

karlnapf commented Jul 31, 2018